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Abstract 

OEMs  and  operators  of  complex  mission/safety  critical 
systems  are  faced  with  the  requirement  to  mitigate  design 
and  performance  risks  and  their  economic  consequences.  A 
key  issue  for  any  engineering  organization  is  the  integrity  of 
the  analysis  that  is  used  to  support  significant  commercial 
decisions.  Analysis  outputs  used  to  establish  or  validate 
performance  criteria  should  have  an  appropriately  high  level 
of  confidence  associated  with  them  when  entering  into 
significant  financial  contracts.  While  risk  assessment 
methods  and  techniques  for  analysis  are  well  defined  and 
understood  and  are  captured  in  various  international  military 
and  commercial  standards,  the  issue  of  analysis  quality  has 
traditionally  been  neglected  and  is  not  adequately  covered  in 
most  commercially  available  engineering  analysis  tools. 
The  quality  of  data  inputs  determines  the  quality  of  analysis 
outputs.  A  key  factor  is  the  source  of  the  parameters  used  in 
an  analysis.  For  example  input  data  may  be  sourced  from 
operational  data,  or  may  be  based  on  the  engineering 
judgement  of  an  individual  or  a  third  party  organization. 
This  paper  outlines  an  approach  to  analysis  quality 
assessment  in  a  model  based  engineering  environment, 
focusing  on  the  sources  of  data  and  ancillary  information  to 
generate  an  Analysis  Quality  Index  (AQI)  for  the  analysis. 
The  AQI  is  generated  as  a  dashboard  reporting  function  for 
the  engineering  model  that  is  used  to  provide  a  confidence 
rating  on  the  analysis  outputs.  Analysis  Quality  Index 
capability  was  incorporated  into  Maintenance  Aware  Design 
environment  (MADe)  software,  an  integrated  tool-set  that 
combines  engineering  risk  analysis  capabilities  to  support 
systems  engineering,  design  and  through-life  support. 

1.  Introduction 

Risk  management  has  become  a  hot  topic  over  the  last 
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decade,  its  ever  increasing  application  to  engineering 
systems  is  not  always  driven  by  purely  technical 
considerations  (Ross,  K.,  &  Main,  B.W.  (2001)).  Factors 
like  compulsory  compliance  with  standards  (MIL,  ISO)  and 
regulation  (e.g.  FAA),  risk  of  litigation  and  thus  possible 
audits  of  the  risk  assessment  process,  reliability  dependent 
insurance  costs,  changes  in  system  management  approaches 
(Product  Life  Management  (PLM),  Life  Cycle  Management 
(LCM)),  changes  in  sustainment  of  technical  systems 
(Performance-based  Contracts  (PBC)),  risks  to 
environmental  safety  etc.  cause  increased  awareness  that 
failures  of  engineering  assets  can  have  penalties. 

Operation  of  an  engineering  system  inevitably  leads  to 
system  degradation  or  failure  of  various  degrees,  which 
generate  financial,  operational  (ceased  function  of  the 
system)  and  physical  risks  to  assets,  human  operators  or  the 
environment. 

To  deal  with  these  issues  a  range  of  methodologies  have 
been  proposed  and  accepted,  especially  in  the  military 
sector,  there  are  over  150  methodologies  dealing  with  risk 
management  in  engineering  systems. 

The  process  of  risk  management  is  a  two-step  process: 

•  Formalized  risk  identification  using  various 

methodologies  of  risk  analysis  -  Failure  Mode  and 
Effects  Analysis  (FMEA),  Failure  Mode,  Effects  and 
Criticality  Analysis  (FMECA),  Reliability  Block 

Diagram  (RBD),  Fault  Tree  Analysis  (FTA),  etc.  see 
International  Standards  Organization  (ISO)  (2004). 

•  Risk  elimination  by  changes  in  system  design, 

maintenance,  operation  etc. 

Of  course  we  must  remember  that  risks  are  assessed  and 
dealt  with  during  the  design  process,  albeit  not  necessarily 
using  formalized  methods. 

The  objective  of  risk  identification  is  to  determine  how  the 
system  may  fail,  and  how  such  failure  affects  system  safety, 
performance,  availability,  etc.  Analysis  provides  metrics  of 
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risks  (e.g.  criticality,  reliability)  which  are  the  basis  for 
corrective  actions  (e.g.  design  changes  or  changes  in 
maintenance  procedures).  The  formalized  risk  identification, 
depending  on  how  and  when  it  is  applied,  has  varying 
impact  on  risk  reduction.  Ideally,  it  should  be  concurrent 
with  design  of  the  system  so  risks  identified  during  design 
process  can  be  eliminated  and/or  minimized  by  modification 
to  design.  This  approach  is  optimal  in  terms  of  cost,  time 
and  degree  of  risk  reduction. 

However  in  practice,  formalized  risk  identification  (FMEA, 
RBD,  FTA)  is  not  conducted  concurrently  with  design  or  is 
carried  out  too  late  to  accommodate  design  changes.  In 
these  circumstances,  risk  analysis  has  only  limited  impact 
on  the  system  design  and  is  often  conducted  at  completion 
of  the  design  process  to  generate  contractual  deliverables  or 
achieve  compliance. 

The  ‘concurrent  with  design’  approach  is  also  not  possible 
when  dealing  with  legacy  systems.  In  the  case  of  such  a 
system,  we  may  only  use  workaround  solutions  to  mitigate 
risk  (better  maintenance,  sensing)  as  design  changes  are 
often  not  feasible  or  possible.  Methods  like  Reliability 
Centered  Maintenance  (RCM),  Maintenance  Effectiveness 
Review  (MER)  and  Back-fit  RCM  are  used  to  determine 
maintenance  practices  which  can  reduce  operational  risk. 
These  methods  often  lead  to  outcomes  such  as  Condition 
Based  Maintenance  (CBM)  and  Prognostics  and  Health 
Management  (PHM). 

With  a  growing  importance  of  risk  management 
methodologies,  the  quality  of  the  methods  is  becoming 
important.  Low  quality  of  risk  assessment  may  increase 
rather  than  decrease  the  cost  of  designing  and  operating  of 
technical  systems. 

According  to  a  Google  search,  the  topic  of  quality  of  risk 
assessment  is  very  prevalent  -  60,000k  results  for  “risk 
analysis  engineering”  and  81,200k  results  for  “quality  of 
risk  analysis”  engineering  -  it  is  currently  seen  as  an 
important  attribute  of  risk  management.  Table  1  presents  the 
most  widely  used  methods  of  risk  assessment: 

2.  The  problem  -  Current  Approaches  that  impact 

THE  QUALITY  OF  RISK  ANALYSIS 

The  current  industry  approaches  to  support  risk  analysis  are 
primarily  database  or  spreadsheet  based  software.  The  use 
of  such  software  to  conduct  the  required  analysis  generates  a 
number  of  significant  issues  in  terms  of  the  cost  of 
conducting  analysis,  quality  of  the  analysis,  system  level 
analysis  and  scheduling  (Bednarz  &  Marriott  (1988),  Kara- 
Zaitri  C.,  Keller  A.,  Barody  I.  &  Fleming,  P.  (1991), 
Ormsby  A.,  Hunt  J.  &  Lee  M.  (1991).  The  main  factors 
impacting  the  quality  of  analysis  are  the  quality  and  quantity 
of  data  used. 


•  Limited  knowledge  capture  /  reuse 

Spreadsheets  are  an  obstacle  to  knowledge  transfer  which 
impacts  the  quantity  of  data  available  for  risk  analysis.  The 
fact  that  spreadsheets  can  normally  only  be  updated  by  the 
people  that  created  them,  is  also  critical  to  ensure  maximum 
coverage  of  the  risk  analysis.  Spreadsheets  are  not  easily 
configuration  managed  based  on  operational  data  or  as 
changes  in  the  platform  are  made.  Furthermore,  the  results 
of  a  performed  analysis  cannot  be  automatically  transferred 
and  used  to  support  related  analysis  methods. 


Table  1.  List  of  the  most  widely  used  methods  of  risk 
assessment  according  to  Google  search  results  (June  2014) 


Method  of  risk  assessment 

Quantity 

FMEA 

3,270k 

Reliability  Diagram 

20,600k 

Fault  Tree 

30,000k 

Fault  Analysis 

45,200k 

Failure  Analysis 

113,000k 

Performance  Based  Contract 

70,200k 

Engineering  Risk  Audit 

34,400k 

Condition  Based  Maintenance 

16,700k 

•  Inconsistency  of  terminology 

The  quality  in  the  analysis  is  significantly  impacted  by  the 
lack  of  industry  wide  taxonomies  to  define  functions  and 
failure  concepts,  which  brings  issues  of  ambiguity  and 
inconsistency  of  terminology.  Risk  analyses  are  also  artefact 
driven  (based  on  attributes  of  the  platform)  and  performed 
on  a  specific  state  of  the  system.  A  snapshot  of  the  system  is 
thus  captured  by  the  analysts  in  spreadsheets  and  the 
designer  is  rarely  involved  in  all  iterations  of  the  analyses  - 
this  can  lead  to  poor  data  quality  that  is  used  in  the  risk 
analysis. 

•  Retrospective  analysis 

Usually  analysis  is  done  retrospectively  (rather  than 
concurrently)  at  the  end  of  the  design  process  using 
spreadsheets/database  FMEA/FMECA  mainly  to  document 
the  outcomes  for  compliance  or  contractual  requirements. 
Evans  J.  (1992)  in  his  editorial  wrote  “..The  idea  that  all  the 
experts  and  number-crunchers  should  come  in  after  a  design 
was  virtually  complete,  and  second-guess  the  designers  was 
stupid  to  begin  with..”. 

•  Disparate  models 

Industry  practice  usually  relies  on  the  usage  of  disparate 
models  of  a  platform  and  its  Bill  of  Materials  (BOMs)  that 
reside  within  the  functional  stovepipes  of  an  organization. 
This  is  an  obstacle  for  comparing  and  controlling  the  data. 
Inconsistencies  in  models  such  as  holes  in  the  BOMs  or  in 
the  structure  of  the  system  may  cause  coverage  losses  that 
are  not  obvious  using  spreadsheets. 
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•  Bottom-up  -  inductive  approach. 

Current  methods  to  conduct  risk  analysis  are  inductive 
(based  on  brainstorming)  and  use  a  bottom-up  approach.  It 
is  therefore  difficult  to  visualize  and  aggregate  all  the  data 
in  order  to  analyze  a  system  in  whole.  Each  piece  of  the 
system  data  is  stored  by  each  stakeholder  in  spreadsheets. 
This  implies  a  suggestive  process  to  support  the  risk 
analysis  process  as  the  assumptions  underlying  analysis, 
data  sources  and  knowledge  of  thought  processes  of  the 
team  members  are  generally  not  recorded.  As  a  result,  the 
quality  and  coverage  are  affected:  a  bottom-up  approach 
may  result  in  comments  being  missed  (coverage)  and 
missing  the  source  of  the  data  (brainstorming). 

•  Subjective  analysis  audit 

Various  FMEA  guides/books  stress  the  importance  of 
FMEA  quality  see  Carlson.  C.  S.  (2012)  and  McKinney  B. 
(1991).  However,  the  FMEA  quality  audit  is  rather 
subjective  as  it  relies  on  subject  matter  expertise  and  often  is 
limited  to  checking  that  the  standard  procedure  was 
correctly  followed.  This  does  not  provide  accurate  and 
objective  assessment  of  the  quality  of  analysis.  A  major 
problem  is  repeatability  of  FMECA  when  carried  by  a 
different  team  of  analysts  (Bell  D.,  Cox  L.,  Jackson  S.  & 
Schaefer,  P.  (1992)). 

•  Platform  reliability  based  on  design  parameters 

In  current  engineering  practices,  designers  do  not 
necessarily  understand  how  the  operators  will  use  the 
system  and  this  is  a  critical  issue  for  the  reliability  of  the 
platform  as  (Reliability,  Availability  and  Maintainability) 
RAM  /  (Integrated  Logistics  Support)  ILS  should  be  based 
on  operationally  determined  RAM  parameters  rather  than 
the  design  parameters.  Design  parameters  are  normally 
sourced  from  third  party  references  that  do  not  account  for 
concept  of  operations,  environment,  etc.  Thus  it  is  important 
to  document  the  source  of  the  information,  and  list 
associated  assumptions  or  else  quality  issues  will  occur. 

•  Isolated  system  analysis 

Historically  individual  technical  risk  assessments  associated 
with  the  deferral  of  maintenance  or  acceptance  of  technical 
defects  are  conducted  in  isolation  using  spreadsheets  and 
therefore  do  not  take  into  account  the  potential 
dependencies  across  the  platform.  This  could  lead  to  either 
safety  issues  or  equipment  breakdown  and  thus  additional 
efforts  to  mitigate  risk.  Integrating  isolated  analysis  on  the 
higher  system  level  by  merging  different  spreadsheets  is 
almost  impossible  due  to  potential  taxonomy  and  hierarchy 
issues.  This  impacts  the  quality  of  the  aggregated  analysis 
performed  at  the  system  level. 

3.  Maintenance  aware  design  environment  (made) 

MADe  (Rudov-Clark  S.,  Stecki  J.  &  Stecki  C.  (2011))  is  a 
model-based  engineering  software  tool  for  conducting  risk 


assessment  (FMECA,  RAM,  RCM,  FTA)  -  where  each 
element  in  the  model  is  associated  with  a  number  of  key 
attributes  such  as  its  functional  description,  the  specific 
physics  of  failure  information  (cause,  mechanism,  fault, 
symptoms)  -  as  shown  on  Figure  1-  and  their  relevant 
criticality  based  on  the  system  performance  requirements. 

▼  ▼  ▼ 

Insufficient  lubricant  Solid  partide  Insuffident  dearances 

(Poppet_Body)  contaminants  (Poppet_Body) 


Surface  change  Guide  Poppet 

Displacement 

(Poppet_Body) 


Figure  1 .  MADe  Failure  diagram  -  mapping  of  failure 
concepts 

MADe  utilizes  simulation  to  propagate  and  trace  the 
dependencies  and  impacts  of  any  fault  injected  into  the 
system  as  shown  on  Figure  2.  This  data  is  used  to  generate  a 
functional  risk  assessment  based  on  the  associated  physics 
of  failure.  Simulation  is  an  important  feature  of  the  tool,  as 
with  highly  complex  systems  it  is  difficult  to  identify  how 
the  impacts  of  a  failure  will  propagate  -  without  this 
knowledge  it  is  impossible  to  accurately  determine  the 
criticality  of  a  specific  failure  mode. 

MADe  automates  the  dependency  mapping  of  a  system 
using  the  functional  path  propagations  that  are  generated  in 
the  model.  The  system  model  is  easily  updated,  modified 
and  MADe  enables  to  conduct  4what-if  analysis  for  an 
actual  or  proposed  design  and  its  constituent  systems, 
components  and  parts. 

As  it  is  simulation  based,  the  software  is  fundamentally  and 
significantly  different  from  spreadsheet/databased  tools 
because  the  model  and  therefore  the  analysis  is  extensible, 
objective  and  repeatable.  As  a  Model  Based  Engineering 
(MBE)  tool,  MADe  offers  a  number  of  advantages  over 
available  spreadsheet/database  FME A/RAM  toolsets. 

•  Knowledge  capture,  reuse  and  transfer 

All  knowledge  about  the  system  and  its  components  is 
captured  in  models  which  can  be  saved  and  reused  for  any 
other  project.  These  user  developed  models  are  stored  in  a 
re-usable  directory  called  a  Library. 
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Figure  2.  MADe  functional  diagram  of  landing  gear  -  showing  failure  propagation 


Models  of  components/systems  can  be  loaded  from  the 
Library  and  re-used  to  represent  a  new  system 
(dependencies  will  be  automatically  established).  The  key 
benefit  is  the  improved  quality  of  analysis,  as  knowledge  is 
captured  and  re-usable  for  future  projects. 

•  Standardized  taxonomy 

MADe  uses  standardized  taxonomy  of  functions/failure 
concepts  to  ensure  that  there  is  consistency  of  terminology 
(and  therefore  understanding)  within  the  organization  and 
currency  of  data  at  each  stage  of  the  platform  life -cycle 
(Rudov-Clark,  S.D.  &  Stecki,  J.  (2009). 

Audit  and  validation  are  based  on  the  input  of  references  for 
the  sources  of  data.  A  standardized  taxonomy  brings 
objectivity  in  the  performed  analysis. 

•  Concurrent  engineering 

Model-based  Engineering  (MBE)  enables  concurrent 
engineering  features  such  as  functional  simulation  which 
means  that  the  development  of  a  system  model  can  be 
associated  with  the  functional  requirements  of  a  system 
rather  than  a  specific  design.  This  enables  the  ability  to 
generate  the  model  -  and  conduct  modelling  analyses  -  at  the 
conceptual  stage  of  the  design  process  to  evaluate  the 
impact  of  changes  to  the  design  and  mitigate  risk  at  an  early 
stage  in  the  platform  life-cycle. 

•  Integrated  capabilities 

MADe  uses  a  single  model  (a  Single  Source  Of  Truth 
(SSOT))  as  basis  for  other  analysis  tasks.  A  model  of  the 
system  is  used  for  reliability  analysis  (both  functional  and 
hardware),  sensor  selection  (sensors  coverage),  Reliability 
Centered  Maintenance  (RCM)  etc.  This  eliminates  the  need 
to  export  data  or  results  of  analysis  as  the  same  model  is 
used  for  all  the  analysis. 


•  Configuration  management  of  the  analysis 

Because  MADe  generates  each  analysis  based  on  the 
common  system  model,  the  impacts  of  any  changes  made  by 
other  functional  groups  within  the  organization  are 
automatically  reflected  in  the  model  (and  thus  future 
analyses).  This  considerably  improves  the  quality  of 
analyses  as  data  come  from  a  SSOT  model. 

•  Integrated  system  analysis 

The  toolset  uses  automated  dependency  mapping  which 
eliminates  the  manual  determination  of  the  impacts  of 
failures  across  the  system.  This  enables  risk  analysis  to  be 
based  on  objective  and  verifiable  data.  MADe  automatically 
establishes  these  connections  and  updates  them  when  the 
system  model  is  modified.  This  is  a  major  benefit  for 
increasingly  complex  and  integrated  systems.  The  level  of 
details  and  dependency  mapping  enable  risk  identification  at 
the  platform  level  down  to  the  component  level  leading  to 
enhanced  traceability  of  data. 

•  Dependencies  mapping 

A  functional  model  represents  a  flow  of  energy,  material  or 
signal  in  the  system.  Based  on  (SSOT)  model  of  the  system, 
functional  relationships  and  failures/effects  dependencies  in 
a  system  for  both  functional  and  physical  failures  are 
defined  using  standardized  taxonomies. 

•  What  if..”  and  “As  is..”  analysis 

“What  if...”  analyses  are  often  focused  on  the  rearrangement 
of  connections  between  models  and/or  inclusion  of  different 
components.  This  capability  is  normally  too  time  consuming 
to  be  achieved  using  a  database  approach,  but  can  be 
expedited  using  a  MBE  approach  (e.g.  copy -paste  and 
library  re-use)  leading  to  otherwise  unachievable  options. 
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MADe  has  the  ability  to  update  the  parameters  in  the  model 
based  on  operational  data  in  order  to  conduct  analysis  of  the 
system  based  on  an  ‘as-is’  performance  state  rather  than 
‘expected’  (design)  state.  This  has  a  significant  impact  on 
the  supportability  posture  for  equipment. 

•  Objective  analysis  audit 

An  objective  approach  to  conduct  risk  analysis  is  beneficial 
for  audit  purposes  and  quality  checking.  A  good  example  of 
efficient  risk  analysis  verification  is  FMECA.  Using  an 
AQI,  the  analyst  can  easily  check  the  completeness  of  the 
analysis  based  on  the  quality  and  quantity  of  the  data  inputs. 
When  it  comes  to  project  management,  an  AQI  can  provide 
a  means  to  evaluating  the  confidence  level  of  a  system 
globally  or  a  particular  risk  analysis  in  order  to  validate  a 
project. 

•  Effective  integration  with  the  organization  IT 

architecture  (specifically  PLM). 

Current  challenges  in  PLM  consist  in  using  a  single  point  of 
truth  for  the  RAM  /  ILS  analysis  that  can  be  shared  by  the 
design  /  supportability  engineering  communities.  As  a 
simulation  based  model,  MADe  offers  the  ability  to 
configuration  manage  the  associated  ILS  analysis  and 
outputs  for  a  system  and  automatically  regenerate  the 
artefacts  that  result  from  any  modification  to  the  design  or 
changes  to  the  maintenance  regime. 

4.  Analysis  quality  assessment 

For  any  analysis  or  simulation  based  analysis,  poor  quality 
inputs  or  improperly  defined  scenarios  create  meaningless 
results.  How  then  to  assess  the  quality  of  risk  analysis? 

Analysis  Quality  Index  (AQI)  is  the  process  of  determining 
that  an  analysis  provides  a  correct  outcome  or  solution.  An 
AQI  may  be  applied  to  numerous  different  analyses  or 
algorithms  (e.g.  FMEA,  Criticality,  Reliability)  to  evaluate 
and  document  the  accuracy  of  the  results.  An  AQI  process  is 
implemented  in  MADe  to  increase  data  quality  and  enable 
objective  audit  of  risk  analysis.  The  main  function  of  an 
AQI  is  to  enable  the  modeler  to  capture  the  assumptions 
used  during  the  process  of  creating  the  model.  A  work  flow 
assessing  an  AQI  is  shown  in  Figure  3.  In  MADe  the 
process  starts  with  setting  up  annotation  policy  Figure  4. 

The  findings  from  an  AQI  can  be  used  to  document  an 
analysis  or  query  the  effectiveness  of  another  analysis.  An 
example  of  this  is  performing  an  AQI  on  a  FMEA  to 
determine  the  confidence  of  a  particular  subsystem,  which 
when  integrated  to  the  system  level  can  identify  high-risk 
areas  in  a  project. 

When  carrying  out  engineering  functions,  assumptions  may 
not  be  listed,  or  listed  after  the  fact  leading  to  poorly 
documented  work. 
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Figure  3.  AQI  workflow  implemented  in  MADe 


Figure  4.  Annotation  policy  setting 
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The  quality  of  the  assumptions,  data  and  parameters  used  in 
a  model  directly  affects  the  integrity  of  any  analysis  output. 
The  solution  for  this  issue  in  MADe  a  user  enters 
assumptions  for  each  piece  of  data,  Figure  5. 


Iuff030811up  -  Annotations 


However,  it  is  difficult  to  keep  track  of  the  data  sources  and 
assumptions  that  support  any  parameter  used  in  a  model, 
particularly  if  multiple  stakeholders  (including  departments, 
groups,  teams  and  external  suppliers)  are  involved  in  system 
development.  Therefore  a  structured  approach  to 
documentation  and  assessment  of  data  quality  is  essential. 

Considering  the  evolutionary  nature  of  a  model,  it  becomes 
necessary  to  capture  this  information  concurrently  as  the 
user  is  modelling.  Using  this  facility  will  allow  more 
accurate  models  based  on  listing  of  the  relevant 
assumptions,  detailed  entries  including  narratives  and  more 
consistent  processes  by  capturing  considerations.  Shown  in 
Figure  6,  each  parameter  edited  or  changed  in  the  model  can 
be  tracked  and  assessed  using  an  annotation  feature  that 
requires  each  stakeholder  to  document  his  data. 


To  summarize  the  data  quality  assessment  of  a  model-based 
risk  analysis  such  as  FMEA/FMECA,  requires  evaluation  of 
two  key  metrics: 

•  Completeness  of  Data  (Data  Coverage) 

•  Data  Quality 

Once  those  two  metrics  are  assessed,  they  can  be  aggregated 
to  determine  the  overall  confidence  level  of  a  particular  risk 
analysis  or  completeness  of  a  model.  An  AQI  becomes 
increasingly  important  as  the  analysis  or  models  become 
more  complex,  thus  requiring  greater  control  and 
management  of  a  larger  set  of  data.  The  quality  assessment 
concept  is  especially  beneficial  for  model-based  risk 
analysis. 

4.1.  Data  Coverage 

The  AQI  is  a  metric  that  may  be  used  to  determine  the 
completeness  of  data  used  in  the  analysis.  Missing  data 
regarding  the  system  can  result  in  poor  coverage  of  the  risk 
analysis,  especially  in  a  complex  analysis  where  there  are 
numerous  inputs  required.  If  any  of  these  inputs  are  missing 
then  the  completeness  of  the  analysis  is  weakened. 
Completeness  can  be  considered  as  the  ratio  the  amount  of 
data  entered  /  the  amount  of  data  required.  Therefore  if  all 
data  for  a  process/analysis  is  entered  then  the  completeness 
would  be  100%,  providing  a  high  confidence  with  the 
process/analysis.  A  higher  completeness  will  improve 
confidence  during  an  audit  and  prove  better  traceability  of 
the  analysis.  Although  it  is  important  to  note  that  while  an 
analysis/process  is  complete,  it  may  not  be  high  quality. 

4.2.  Data  Quality 

Data  quality  involves  documenting  the  source,  confidence 
level  and  assumptions  underlying  each  piece  of  data  that  is 
used  as  an  input  parameter  for  the  analysis. 

This  process  aims  at  documenting  critical  questions 
regarding  a  model  or  a  particular  analysis: 

•  Where  does  a  particular  parameter  or  data  set  come 
from? 
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Model  Annotations 


Status:  |^j  Pending  ig£i  Annotated  1^41  Unchanged 
Item/Event  Description:  Search... 
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MTTF  (hrs)  changed  from  12360.9  to 
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Edited  ^  Required  Mechanism  -  Buildup  of  debris  (Speed  Sensor)  Difficulty  of  Detection  changed  from  10.0  to  3.0  Engineer  Development  data  2014/04/24  15:50:48 


(  Annotated  )  2014/04/24  15:36:30 


.Required  Cause  -  Liquid  contaminant  (Speed  Sensor)  Difficulty  of  Detection  changed  from  1 0X>  to  3j0  Engineer  Development  data  2014/04/24  15:50:48 


Figure  6.  Annotation  summary 


558 


Annual  Conference  of  the  Prognostics  and  Health  Management  Society  2014 


•  Who  sourced  this  data? 

•  Why  was  it  set  to  this  particular  value? 

•  Which  confidence  to  assign  to  a  particular  data? 

The  quality  of  data  can  range  from  conceptual 
(brainstorming)  to  collected  data  (operation)  and  is 
important  in  defining  the  quality  of  the  data  used  in  the 
analysis.  Previous  articles  on  the  quality  of  analysis  (Evans 
J.  (1992).)  explain  that  in  order  to  avoid  poor  data  quality, 
“it  is  essential  for  everyone  with  a  real-word  problem  to 
insist  on  an  adequate,  numbered,  list  of  assumptions,  where 
the  assumptions  are  in  reasonably  plain  language”.  To  rank 
quality,  different  categories  can  be  assigned  which 
correspond  to  different  sources  (e.g.  engineer,  database, 
etc.).  By  defining  a  data  source  type,  a  confidence  level  can 
be  assigned  to  each  type  which  may  be  aggregated  to 
provide  an  overall  level  of  confidence.  As  the  quality  of  the 
data  sources  increases  so  does  the  quality  of  the  analysis. 
The  categories  and  weightings  of  sources  can  be  adjusted 
for  specific  environments  or  applications.  It  is  also 
important  to  track  the  source  where  data  is  obtained  from, 
note  the  source  of  the  information,  time/date  of  data  entry 
and  allow  annotation  of  a  particular  entry.  This  information 
is  automatically  updated  as  data  is  being  annotated  in  the 
model  to  provide  the  percentage  of  annotated  data,  data 
quality,  as  well  as  an  overall  confidence  level  in  the  model 
as  shown  in  Figure  7  and  Figure  8. 


Figure  7.  Coverage,  quality  and  confidence  level 


5.  Conclusion 

This  paper  has  outlined  a  unique  approach  to  assess  the 
quality  of  risk  analysis  in  a  model  based  engineering 
environment.  In  current  industry  approaches,  the  extensive 
usage  of  spreadsheet/database  based  tools  to  conduct  risk 
analysis  generates  a  number  of  significant  issues  in  terms  of 
cost  of  conducting  analysis,  quality  and  objectivity  of  the 
analysis,  as  well  as  system  level  analysis.  To  solve  those 
issues,  it  is  essential  to  conduct  data  quality  assessment 
focusing  on  the  quality  and  quantity  of  data  used  as 
parameters  in  the  analysis.  A  good  example  of  assessing  the 
quality  of  analysis  is  to  apply  data  quality  assessment  to 
model-based  risk  analysis.  The  quality  assessment  process 
implemented  in  the  MADe  software  provides  objective 
auditability  of  all  relevant  information  regarding  a  particular 
analysis  or  a  whole  system.  The  confidence  level  in  analysis 
outputs  and  thus  the  quality  of  analysis  are  optimized  by: 


•  Documenting  and  reviewing  all  parameters  used  in  the 
model  /  analysis. 

•  Mitigating  posting  cycle  issues  as  expert  knowledge  to 
a  project  file  is  retained. 

•  Ensuring  that  all  relevant  supporting  assumptions  are 
captured. 


4)  Engineer  ^  Peer  Reviewed  Discussion  4  Published  Database 

£  OEM  £  Operating  Data 

Figure  8.  Pie  chart  showing  origin  of  data 

6.  Future  Work 

While  this  paper  has  focused  on  presenting  the  application 
of  data  quality  assessment  to  a  model-based  risk  analysis 
(AQI)  there  are  other  possible  applications  of  data  quality 
assessment. 

•  Model  Quality  Index  (MQI) 

This  is  the  process  of  assessing  the  manner  and  degree  to 
which  data  used  in  a  model  is  an  accurate  representation  of 
the  real  world  and  of  establishing  the  level  of  confidence  of 
this  assessment.  This  index  would  be  useful  in  model  or 
simulation  environments  to  determine  the  validity  and 
correctness  of  a  model  compared  to  the  system  it  is  based 
upon.  The  findings  from  an  MQI  could  be  useful  in  learning 
how  to  create  a  more  accurate  or  correct  model  of  a  system. 

•  Process  Quality  Index  (PQI) 

This  is  the  process  of  assessing  the  confidence  and 
adherence  to  a  particular  workflow  or  process.  This  could  be 
applied  to  an  engineering  process  and  used  to  assist  learning 
of  a  new  process  or  even  the  audit  of  an  existing  process 
within  a  company.  Findings  from  a  PQI  could  be  applied 
back  into  the  process  to  optimize  it  for  its  function  within  a 
company. 
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